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abstract 

The purpose of this 1970-1971 evaluation was to 
examine the degree to which this cluster of projects has been 
contributing to the attainment of the community— related goals of the 
School District, and to test the validity of the assumptions (on 
which these projects are based) that information leads to 
participation and that information and participation have a positive 
effect on the opinions of school and community about each other. 
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FOREWORD 



School districts, the federal government , other funding agents, 
the community, and parents are becoming increasingly concerned with the 
impact of resource expenditures on educative outcomes. Historically, 
resource funds in excess of central school budgets have taken the form of 
special prograius/pro j ects designed to meet the needs of pupils within 
designated populations of a school district- Although these expenditures 
have produced significant educational changes for the designated popula- 
tion, parents and the community have requested that the level of addi- 
tional funding be made available for all pupils in the entire school popu- 
lation. However, none of the proposed methods for securing additional 
funds (e.g., taxes or bond issues) has been successful in meeting these 
increasing demands. 

Since additional monies have not been made available to central 
school budgets, more extensive monitoring and substantiation procedures 
have been introduced by federal and private funding agencies to ascertain 
whether the recipients are using the appropriated/contributed funds in 
accordance with the specified guidelines, and (b) whether the school sys- 
tem has made in— kind contributions and subsequent programmatic/ instruc- 
tional changes as a result of the funding. The emphasis on the use of 
cost-accounting or cost-effectiveness procedures has been instituted as a 
means for determining whether a given program/project is producing out- 
comes that are better than or equivalent to practices costing more or 
less per unit. The intent of such analyses is to provide information that 
could assist school management and funding agencies in making judicious 
decisions concerning the feasibility of continuing and/or expanding iden- 
tified successful efforts into other schools within a given school dis- 
trict or into other districts within the country. 

Current methods of evaluation have demonstrated that the tech- 
niques employed to gather information about the attainment of specified 
behavioral objectives are effective. However, these methodologies have 
not demonstrated their capabilities to explain why certain programs/pro j — 
ects are not successful (a) upon replication within a given school or 
(b ) in those situations where the programs/pro j ects have been expanded. 
Longitudinal data on projects implemented in specific schools reveal that 
such projects are usually successful for those who are directly involved 
in them- Moreover, since some of these programs/pro j ects are specific 
for certain grades, students are usually successful performers in the 
grades in which the programs/pro j ects are active, but tend to return to 
their previous pattern of underachievement in subsequent years. 

This sporadic achievement pattern of children who move into and 
out of special programs/pro j ects is of special concern to school dis- 
tricts, school administrators, parents, and the community since it is they 
who have the responsibility for the whole educational experience of each 
child. Parents and the community are particularly dismayed and do not 
understand why sustained improvement cannot be achieved throughout the 
entire educational experience of their children. 





School districts like Philadelphia's have employed a cadre of 
highly specialized, competent educational researchers who perform active 
and comprehensive evaluation services for the school districts. It i^ to 
these professionals that the school management, teaching staff, community, 
parents, and students turn for reliable information which could be used to 
improve the educational programs. Although such professional evaluation 
staffs have provided reliable information relative to the specific pro- 
gram s/pro j ects within the district, they have not been actively gathering 
information that could assist in answering broad comprehensive questions 
upon which major decisions are made. Therefore, gaps appear in informa- 
tion obtained through individual project evaluations and tend to raise 
doubts about the capability of such professionals to produce reliable in- 
formation. Consequently, the full impact of the evaluation staff's capa- 
bility to assist in the decision-making process of school districts has 
not been realized. 

Although it is recognized that students directly involved in 
special programs/projects receive inestimable benefits from the experi- 
ence, many questions have been raised concerning the worth of such expen- 
ditures to students not involved in the experience. In another form, the 
same concern is exhibited by the desires of some to dissolve the special 
programs for more global experiences — that is, situations that have direct 
applicability to a larger number of students and teachers - 

In the face of these considerations, the initiation of major 
policies for systemwide changes requires at least three kinds of informa- 
tion about individual programs /pro j ects and combinations thereof: (a) the 

extent of their total impact in the School District, (b) why they work 
successfully in some schools and not in others, and (c_) whether their 
current implementation represents an optimal or unique programmatic en- 
vironment which produces success for a specified student population. 

The need to provide such information has caused the Department 
of Instructional Systems Research to change its ESEA, Title I evaluation 
strategy for the 1970—1971 school year. Instead of concentrating its 
efforts on determining whether each Title I project, was reaching its in- 
dividual goal, an attempt was made to ascertain the impact of the Title I 
projects in their particular service, programmatic, or supportive areas. 

It is believed that the Department of Instructional Systems Research can 
undertake this new emphasis because it has developed, over the past three 
years, a firm project performance base line — that is, the characteristic 
outputs of given Title I projects have been demonstrated- The consistent 
findings for each project suggest that the implementation procedures es- 
tablished for each project are producing their maximum service output. 
Moreover, it appears that the only ways in which the independent projects 
could raise their present level of output would be through (a ) higher 
levels of resource funding or (b) improvements in the managemer.it of in- 
structional knowledge and practices. 

Of the two alternatives mentioned, the improvement of instruc- 
tional management techniques (delivery of instructional services) seems 
more tenable. However, the Department of Instructional Systems Research 
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has not been gathering systematic data which could answer broad educa- 
tional management questions. For example, "Can pupil achievement output: 
be improved by a particular mix or combination of Title X projects? Khat 
S02ne the school characteristics which seem to maximize the impact 
potential of the instructional programs sponsored by both central and 
federal budgets? Are the program: ./pro j ects presently budgeted and imple- 
mented helping to achieve the School District's overall objectives?" 



To secure such information, a method for obtaining information 
a ^ >out the impact of Title X projects on the broad operational goals of the 
School District has been undertaken. This technique has been called 

e v aluation . in the execution of this evaluation technique, the 
major emphasis is upon determining (a) whether the common objectives of 
the clustered projects are being achieved, (b> whether the clustered 
projects are contributing to the attainment of the School District's 
goaj_s , and ( cO whether there are requisite conditions which insure the 
success of each individual project within the cluster (be they school- 
specific or project-specific) . 

The cluster approach also permits the gathering of evaluative 
information to meet the specifications of Title X guidelines and affords 
the evaluator an opportunity to observe and investigate the project's 
impact from another vantage point. For example, instead of attempting to 
determine whether the School— Community Coordinator project is meeting its 
specific objectives, as was done for years 1968 through 1970 , a broader 
investigation was conducted to ascertain (a) whether the goal of improving 
community participation in school -related activities was being realized, 
and (b) whether the assumptions and thrusts under which the project is 
operating lead to the successful attainment of these goals. In question 
form. Are the current implementation methods and procedures used in proj- 
ects having school -community goali; motivating parents to became more in- 
volved in school-community activities?" Like emphases have been 
articulated in three other central areas: the improvement of instruction- 

al practices , the improvement of students ' knowledge and competence in 
vocational and career selection, and the improvement, of students' cultural 
knowledge, awareness, and social interactions. 

It is the belief of the Director of the Division of Instruc- 
tional Research and Development that the proposed emphasis upon the 
cluster-evaluation approach will provide (a) more comprehensive informa- 
tion for all levels of school management, the community, teachers, and 
students concerning the impact of the Title I project offerings on the 
attainment of articulated goals and concerns of the School District, and 
(b) xnora interactive information for project managers who are responsible 
for improving the capacity of the individual projects to deliver the ser— 
vices for which they were organized. 



Edward K . Brown , 

Director , 

Division of Instructional 
Research and Development 
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EVALUATING TITLE I ESEA PROJECTS IN CLUSTERS 



This volume culminates an intensive cooperative effort by the 
Department of Instructional Systems Research and. other persons in the 
Division of Instructional Research and Development 1 to look at Philadel- 
phia " s ESEA, Title I program as a whole, greater than the sum of its 
separate parts . 



A Broader Approach to Project Evaluation 

The reorganization in 1970 of what was then called the Depart- 
ment of Program Assessment made it appropriate to rethink the project-by- 
project evaluation methodology employed during the two preceding years. 

though useful for producing base-line data, the individual 
project technique had certain fundamental limitations. Since students 
were exposed to several projects simultaneously, individual project 
evaluations might have provided questionable information (e . g . , whether 
the children gained because of Project X or because of Project y) . An 
examination of the relationships existing between groups of projects and 
the attainment of School District goals seemed to be more relevant. 
Across-pro ject evaluation could ascertain what changes were produced and 
might identify aspects of the total effort calling for revision. From 
such analyses, evaluation findings might identify outcomes which were con 
tinge nt upon particular antecedent conditions and project processes . 

Assessments of the impact of the Title I projects were envi- 
sioned as in formation ■-'System networks which could provide data that met 
the specifications of the Elementary end Secondary Education Act's guide- 
lines find at the same time would be consistent with the needs of the 
School District. Methods and procedures ware designed to gather relevant 
information concerning selected impact areas which embraced the needs for 
these two levels of information . To meet the service responsibilities of 
the Department, a holistic approach was initiated. Under this technique, 
personnel were assigned in accordance with specified management— informa- 
tion needs rather than the accumulation of peripherally related projects. 
It was believed that this method of gathering specified clusters" of 
related management information would provide a more viable system for ob- 
taining the variety of information required by various levels of manage- 
ment, and facilitate an in— depth look at more comprehensive evaluation 
questions . 



* Robert Reiter has contributed extensively to report-format 
development, and his editorial assistance has helped to clarify thinking 
and to bridge technical communication gaps. Mrs. Deborah Kraisler and 
Frances Byers have provided timely and extensive data— analysis 
support. 



An examination of their goals revealed that the Title X projects 
in Philadelphia could be grouped into four general programmatic catego- 
ries: 



1- School— community relations and unique staffing patterns 

2 . Instructional practices and student cognitive performance 

3. Auxiliary services to schools and pupils 

4. Other Title I projects. 

School-community relations and unique staffing patterns encom- 
passes those projects which are designed to establish, refine, and 
strengthen the lines of communication between target-area schools and the 
communities they serve in order to facilitate the interpretation of school 
programs to the local area, apprise school personnel of community needs 
and aspirations, develop and implement joint efforts to improve the neigh- 
borhood, and obtain relevant information which will enable schools to 
operate more knowledgeably. 



Instructional practices and student cognitive performance encom- 
passes those projects which are designed (a) to encourage children, 
through diversified instructional materials, to work voluntarily with 
materials and situations in the learning environment, to function effec- 
tively in problem-solving situations , and to communicate their experiences 
orally or in a graphic or narrative display; (b) to provide specialized 
instruction and materials required by children with unusual learning prob- 
lems, such as Spanish-speaking children, children with speech and oral- 
language disorders, and children who are mentally retarded and/or emotion— 
all Y disturbed; and (c_) to integrate appropriate hardware, software, and 
instructional strategies into multimodal approaches to reading, and other 
basic skills, that would tend to reverse the current trend toward drastic 
under achievement in fundamental subjects. 

Auxiliary services to schools and pupils encompasses those proj- 
ects which are designed (a_) to provide special services for underachiev- 
ing students that will facilitate emotional, intellectual, and social 
growth , and encourage potentially able target— area children to continue 
their education after the completion of high school; and (b) to provide an 
opportunity for target— area children to improve their vocational compe— 
tency through instruction in self-chosen vocational areas and to expand 
"their career choices through exploratory and hands-on experiences . 

Other Title I projects are those projects which have as a common 
denominator aspects and concerns of a more general nature and, thereby, 
seek (ci) to provide preschool and kindergarten children with additional 
personnel, activities, and care so that their initial exposure to learning 
will be enriched, thus establishing the foundation for future cognitive 
development by facilitating the development of a positive self-image, a 
sense of belonging, and feelings of security; (b) to provide target-area 
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